Measuring Software Quality: A Case Study

The National Security Agency's mission is to provide support for the security of the United States. Over the years, the Agency has become extremely dependent on the software that makes up its information technology infrastructure. NSA has come to view software as a critical resource upon which much of the world's security, prosperity, and economic competitiveness increasingly rests. If anything, dependence on software and its corresponding effect on national security makes it imperative for NSA to accept and maintain only the highest quality software. Cost overruns, or software systems that are defective or of low quality, can impose a significant burden on national security and NSA's mission. The NSA spends many hundreds of millions of dollars annually on software development and maintenance. An internal NSA study shows that an average software project generates only seven to eight lines of delivered code per person per day, at a cost of approximately $70 per line (assuming a $140,000 annual loaded cost for each software developer). Viewed in these terms, it becomes clear that we need to lower software costs by finding ways to improve productivity while still maintaining quality. NSA is no more immune than the rest of the software industry to the problems of low-quality software. I myself spent the first five years of my contractual relationship with NSA on software projects that were consolidated, cut back, or simply stopped at the government's discretion. Most of the problems on those projects resulted from highly unstable, error-prone software. Why does this happen? It happens because software development is intensely manual in nature. A lot of time and effort goes into coding, and software inevitably falls victim to rushed schedules, constantly changing requirements, poor process, failure to adhere to software engineering practices, and indifferent coding practices and standards. Yet software quality requires a defined process supported by automated technology and the application of a base set of measures. The rub is the difficulty in measuring software especially the specific task-level activities of software development executed by a programmer/engineer. The traditional algorithms and metrics from the hard engineering disciplines either come up short when applied to software or are easily misapplied. Software engineering is a discipline that is largely nonparametric and resistant to traditional modes of analysis. Unquestionably, we need common methods of determining the business value of software. We must also generate and promote the human factors and activity statistics necessary to manage and steer software development. So what is NSA doing about software quality?